Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow to skip cover compilation for small tests #3382

Merged
merged 6 commits into from
Nov 4, 2021

Conversation

arcusfelis
Copy link
Contributor

@arcusfelis arcusfelis commented Nov 3, 2021

This PR addresses "small tests are slooow while on dev machine".

Proposed changes include:

  • --skip-cover flag works now
  • no changes in how CI works

That is finally fast:

./tools/test-runner.sh --skip-big-tests --skip-cover --verbose router:routing

How to test the change:

# that is fast
./tools/test-runner.sh --skip-big-tests --skip-cover --verbose router:routing

# that is two minutes long
./tools/test-runner.sh --skip-big-tests --verbose router:routing

@mongoose-im
Copy link
Collaborator

mongoose-im commented Nov 3, 2021

small_tests_24 / small_tests / e91af83
Reports root / small


internal_mnesia_24 / internal_mnesia / e91af83
Reports root/ big
OK: 1586 / Failed: 0 / User-skipped: 297 / Auto-skipped: 0


small_tests_23 / small_tests / e91af83
Reports root / small


ldap_mnesia_24 / ldap_mnesia / e91af83
Reports root/ big
OK: 1483 / Failed: 0 / User-skipped: 400 / Auto-skipped: 0


dynamic_domains_mysql_redis_24 / mysql_redis / e91af83
Reports root/ big
OK: 2703 / Failed: 1 / User-skipped: 203 / Auto-skipped: 0

service_domain_db_SUITE:db:db_keeps_syncing_after_cluster_join
{error,{test_case_failed,{[<<"example1.com">>,<<"example2.com">>,
               <<"example3.com">>],
              [<<"example1.com">>,<<"example2.com">>,
               <<"example3.com">>,<<"example4.com">>]}}}

Report log


dynamic_domains_pgsql_mnesia_23 / pgsql_mnesia / e91af83
Reports root/ big
OK: 2721 / Failed: 0 / User-skipped: 186 / Auto-skipped: 0


dynamic_domains_pgsql_mnesia_24 / pgsql_mnesia / e91af83
Reports root/ big
OK: 2721 / Failed: 0 / User-skipped: 186 / Auto-skipped: 0


ldap_mnesia_23 / ldap_mnesia / e91af83
Reports root/ big
OK: 1483 / Failed: 0 / User-skipped: 400 / Auto-skipped: 0


dynamic_domains_mssql_mnesia_24 / odbc_mssql_mnesia / e91af83
Reports root/ big
OK: 2749 / Failed: 1 / User-skipped: 186 / Auto-skipped: 0

sm_SUITE:parallel:messages_are_properly_flushed_during_resumption_p1_fsm_old
{error,
  {{badmatch,
     {error,
       {connection_step_failed,
         {#Fun<sm_SUITE.11.118723036>,
          {client,
            <<"alicE_messages_are_properly_flushed_during_resumption_p1_fsm_old_80.242916@domain.example.com">>,
            escalus_tcp,<0.31418.1>,undefined,
            [{username,
               <<"alicE_messages_are_properly_flushed_during_resumption_p1_fsm_old_80.242916">>},
             {server,<<"domain.example.com">>},
             {host,<<"localhost">>},
             {password,<<"matygrysa">>},
             {stream_management,true},
             {stream_id,<<"5018957033f85b94">>}]},
          [{compression,[<<"zlib">>]},
           {starttls,true},
           {stream_management,true},
           {advanced_message_processing,true},
           {client_state_indication,false},
           {sasl_mechanisms,[<<"SCRAM-SHA-256">>,<<"PLAIN">>]},
           {caps,undefined}]},
         {timeout,get_resumed}}}},
   [{sm_SUITE,
      '-messages_are_properly_flushed_during_resumption_p1_fsm_old/1-fun-1-',
      3,
      [{file,"/home/circleci/app/big_tests/tests/sm_SUITE.erl"},
       {line,1274}]},
    {escalus_story,story,4,
      [{file,
         "/home/circleci/app/big_tests/_build/default/lib/escalus/src/escalus_story.erl"},
       {line,72}]},
    {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1783}]},
    {test_server,run_test_case_eval1,6,
      [{file,"test_server.erl"},{line,1292}]},
    {test_server,run_test_case_eval,9,
      [{file,"test...

Report log


pgsql_mnesia_24 / pgsql_mnesia / e91af83
Reports root/ big
OK: 3090 / Failed: 0 / User-skipped: 213 / Auto-skipped: 0


elasticsearch_and_cassandra_24 / elasticsearch_and_cassandra_mnesia / e91af83
Reports root/ big
OK: 1861 / Failed: 0 / User-skipped: 327 / Auto-skipped: 0


pgsql_mnesia_23 / pgsql_mnesia / e91af83
Reports root/ big
OK: 3096 / Failed: 2 / User-skipped: 213 / Auto-skipped: 0

sm_SUITE:parallel_manual_ack_freq_1:resume_session_state_send_message
{error,
  {{assertion_failed,assert_many,true,
     [#Fun<sm_SUITE.26.57441291>,#Fun<sm_SUITE.26.57441291>,
      #Fun<sm_SUITE.26.57441291>],
     [{xmlel,<<"presence">>,
        [{<<"from">>,
        <<"alicE_resume_session_state_send_message_75.523626@localhost/escalus-default-resource">>},
         {<<"to">>,
        <<"alice_resume_session_state_send_message_75.523626@localhost/escalus-default-resource">>},
         {<<"xml:lang">>,<<"en">>}],
        []},
      {xmlel,<<"message">>,
        [{<<"from">>,
        <<"bOb_resume_session_state_send_message_75.466270@localhost/escalus-default-resource">>},
         {<<"to">>,
        <<"alicE_resume_session_state_send_message_75.523626@localhost">>},
         {<<"xml:lang">>,<<"en">>},
         {<<"type">>,<<"chat">>}],
        [{xmlel,<<"body">>,[],[{xmlcdata,<<"msg-1">>}]},
         {xmlel,<<"delay">>,
           [{<<"xmlns">>,<<"urn:xmpp:delay">>},
          {<<"stamp">>,<<"2021-11-03T11:24:35.592813Z">>},
          {<<"from">>,<<"localhost">>}],
           [{xmlcdata,<<"SM Storage">>}]}]},
      {xmlel,<<"message">>,
        [{<<"from">>,
        <<"bOb_resume_session_state_send_message_75.466270@localhost/escalus-default-resource">>},
         {<<"to">>,
        <<"alicE_resume_session_state_send_message_75.523626@localhost">>},
         {<<"xml:lang">>,<<"en">>},
         {<<"type">>,<<"chat">>}],
        [{xmlel,<<"body">>,[],[{xmlcdata,<<"msg-2">>}]},
         {xmlel,<<"delay">>,
           [{<<"xmlns">>,<<"urn:xmpp:...

Report log

service_domain_db_SUITE:db:db_keeps_syncing_after_cluster_join
{error,{test_case_failed,{[<<"example1.com">>,<<"example2.com">>,
               <<"example3.com">>],
              [<<"example1.com">>,<<"example2.com">>,
               <<"example3.com">>,<<"example4.com">>]}}}

Report log


mssql_mnesia_24 / odbc_mssql_mnesia / e91af83
Reports root/ big
OK: 3090 / Failed: 0 / User-skipped: 213 / Auto-skipped: 0


mysql_redis_24 / mysql_redis / e91af83
Reports root/ big
OK: 3101 / Failed: 1 / User-skipped: 230 / Auto-skipped: 0

sm_SUITE:parallel:messages_are_properly_flushed_during_resumption
{error,
  {{badmatch,
     {error,
       {connection_step_failed,
         {#Fun<sm_SUITE.11.118723036>,
          {client,
            <<"alicE_messages_are_properly_flushed_during_resumption_75.614510@localhost">>,
            escalus_tcp,<0.7970.2>,undefined,
            [{username,
               <<"alicE_messages_are_properly_flushed_during_resumption_75.614510">>},
             {server,<<"localhost">>},
             {password,<<"matygrysa">>},
             {stream_management,true},
             {host,<<"localhost">>},
             {stream_id,<<"d8c0272f599a510f">>}]},
          [{compression,[<<"zlib">>]},
           {starttls,true},
           {stream_management,true},
           {advanced_message_processing,true},
           {client_state_indication,false},
           {sasl_mechanisms,[<<"SCRAM-SHA-256">>,<<"PLAIN">>]},
           {caps,undefined}]},
         {timeout,get_resumed}}}},
   [{sm_SUITE,'-messages_are_properly_flushed_during_resumption/1-fun-1-',
      3,
      [{file,"/home/circleci/app/big_tests/tests/sm_SUITE.erl"},
       {line,1230}]},
    {escalus_story,story,4,
      [{file,
         "/home/circleci/app/big_tests/_build/default/lib/escalus/src/escalus_story.erl"},
       {line,72}]},
    {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1783}]},
    {test_server,run_test_case_eval1,6,
      [{file,"test_server.erl"},{line,1292}]},
    {test_server,run_test_case_eval,9,
      [{file,"test_server.erl"},{line,1224}]}]}}

Report log


riak_mnesia_24 / riak_mnesia / e91af83
Reports root/ big
OK: 1707 / Failed: 0 / User-skipped: 328 / Auto-skipped: 0


dynamic_domains_mysql_redis_24 / mysql_redis / e91af83
Reports root/ big
OK: 2704 / Failed: 0 / User-skipped: 203 / Auto-skipped: 0


pgsql_mnesia_23 / pgsql_mnesia / e91af83
Reports root/ big
OK: 3090 / Failed: 0 / User-skipped: 213 / Auto-skipped: 0

NelsonVides
NelsonVides previously approved these changes Nov 3, 2021
Copy link
Collaborator

@NelsonVides NelsonVides left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Excellent, love it 😄

Copy link
Collaborator

@NelsonVides NelsonVides left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wait, no, not approved, now I get an async "kill_process_on_exit" log polluting my shell a few seconds after tests are ran.

@NelsonVides NelsonVides dismissed their stale review November 3, 2021 11:11

Wait, no, not approved, now I get an async "kill_process_on_exit" log polluting my shell a few seconds after tests are ran.

@codecov
Copy link

codecov bot commented Nov 3, 2021

Codecov Report

Merging #3382 (42c06f9) into master (fc29465) will decrease coverage by 0.04%.
The diff coverage is n/a.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #3382      +/-   ##
==========================================
- Coverage   80.72%   80.68%   -0.05%     
==========================================
  Files         397      397              
  Lines       32268    32268              
==========================================
- Hits        26049    26035      -14     
- Misses       6219     6233      +14     
Impacted Files Coverage Δ
src/mod_roster_mnesia.erl 72.72% <0.00%> (-21.22%) ⬇️
src/mod_last_rdbms.erl 96.15% <0.00%> (-3.85%) ⬇️
src/mam/mod_mam_utils.erl 86.21% <0.00%> (-3.53%) ⬇️
src/mam/mod_mam_meta.erl 95.18% <0.00%> (-2.41%) ⬇️
src/mod_last.erl 86.76% <0.00%> (-1.48%) ⬇️
src/mod_bosh_socket.erl 78.57% <0.00%> (-0.32%) ⬇️
src/mod_muc_log.erl 78.11% <0.00%> (ø)
src/global_distrib/mod_global_distrib_receiver.erl 80.00% <0.00%> (ø)
src/ejabberd_s2s_out.erl 63.31% <0.00%> (+0.23%) ⬆️
src/pubsub/mod_pubsub_db_rdbms.erl 95.34% <0.00%> (+0.25%) ⬆️
... and 4 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fc29465...42c06f9. Read the comment docs.

@NelsonVides
Copy link
Collaborator

But it works fine and fast if I just do rebar3 ct, which is honestly what I always do 😬

@arcusfelis
Copy link
Contributor Author

But it works fine and fast if I just do rebar3 ct, which is honestly what I always do 😬

It would have cover enabled, because rebar.config says so.

@NelsonVides
Copy link
Collaborator

But it works fine and fast if I just do rebar3 ct, which is honestly what I always do 😬

It would have cover enabled, because rebar.config says so.

You just unset that flag for rebar.config, with this PR rebar3 ct runs immediately. So just don't know what's wrong with doing it the test-runner way with those async logs 🤷🏽‍♂️

@arcusfelis
Copy link
Contributor Author

kill_process_on_exit was here before, it removes async children processes, started by the test script.
You can see that message now, because the script finishes very quick now. @NelsonVides

@mongoose-im
Copy link
Collaborator

mongoose-im commented Nov 3, 2021

small_tests_24 / small_tests / 2c616cd
Reports root / small


internal_mnesia_24 / internal_mnesia / 2c616cd
Reports root/ big
OK: 1586 / Failed: 0 / User-skipped: 297 / Auto-skipped: 0


small_tests_23 / small_tests / 2c616cd
Reports root / small


dynamic_domains_pgsql_mnesia_23 / pgsql_mnesia / 2c616cd
Reports root/ big
OK: 2721 / Failed: 0 / User-skipped: 186 / Auto-skipped: 0


dynamic_domains_pgsql_mnesia_24 / pgsql_mnesia / 2c616cd
Reports root/ big
OK: 2721 / Failed: 0 / User-skipped: 186 / Auto-skipped: 0


dynamic_domains_mysql_redis_24 / mysql_redis / 2c616cd
Reports root/ big
OK: 2704 / Failed: 0 / User-skipped: 203 / Auto-skipped: 0


ldap_mnesia_23 / ldap_mnesia / 2c616cd
Reports root/ big
OK: 1483 / Failed: 0 / User-skipped: 400 / Auto-skipped: 0


ldap_mnesia_24 / ldap_mnesia / 2c616cd
Reports root/ big
OK: 1489 / Failed: 1 / User-skipped: 400 / Auto-skipped: 0

mod_ping_SUITE:server_ping:server_ping_pong
{error,{{badmatch,[{[<<"localhost">>,mod_ping,ping_response],
          {expected_diff,5},
          {before_story,0},
          {after_story,4}}]},
    [{escalus_mongooseim,post_story_check_metrics,1,
               [{file,"/home/circleci/app/big_tests/_build/default/lib/escalus/src/escalus_mongooseim.erl"},
                {line,74}]},
     {escalus_mongooseim,maybe_check_metrics_post_story,1,
               [{file,"/home/circleci/app/big_tests/_build/default/lib/escalus/src/escalus_mongooseim.erl"},
                {line,51}]},
     {escalus_story,story,4,
            [{file,"/home/circleci/app/big_tests/_build/default/lib/escalus/src/escalus_story.erl"},
             {line,75}]},
     {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1783}]},
     {test_server,run_test_case_eval1,6,
            [{file,"test_server.erl"},{line,1292}]},
     {test_server,run_test_case_eval,9,
            [{file,"test_server.erl"},{line,1224}]}]}}

Report log


dynamic_domains_mssql_mnesia_24 / odbc_mssql_mnesia / 2c616cd
Reports root/ big
OK: 2721 / Failed: 0 / User-skipped: 186 / Auto-skipped: 0


pgsql_mnesia_23 / pgsql_mnesia / 2c616cd
Reports root/ big
OK: 3097 / Failed: 1 / User-skipped: 213 / Auto-skipped: 0

sm_SUITE:parallel_manual_ack_freq_1:resume_session_state_send_message
{error,
  {{assertion_failed,assert_many,true,
     [#Fun<sm_SUITE.26.57441291>,#Fun<sm_SUITE.26.57441291>,
      #Fun<sm_SUITE.26.57441291>],
     [{xmlel,<<"presence">>,
        [{<<"from">>,
        <<"alicE_resume_session_state_send_message_12.909009@localhost/escalus-default-resource">>},
         {<<"to">>,
        <<"alice_resume_session_state_send_message_12.909009@localhost/escalus-default-resource">>},
         {<<"xml:lang">>,<<"en">>}],
        []},
      {xmlel,<<"message">>,
        [{<<"from">>,
        <<"bOb_resume_session_state_send_message_12.860147@localhost/escalus-default-resource">>},
         {<<"to">>,
        <<"alicE_resume_session_state_send_message_12.909009@localhost">>},
         {<<"xml:lang">>,<<"en">>},
         {<<"type">>,<<"chat">>}],
        [{xmlel,<<"body">>,[],[{xmlcdata,<<"msg-1">>}]},
         {xmlel,<<"delay">>,
           [{<<"xmlns">>,<<"urn:xmpp:delay">>},
          {<<"stamp">>,<<"2021-11-03T13:20:12.990364Z">>},
          {<<"from">>,<<"localhost">>}],
           [{xmlcdata,<<"SM Storage">>}]}]},
      {xmlel,<<"message">>,
        [{<<"from">>,
        <<"bOb_resume_session_state_send_message_12.860147@localhost/escalus-default-resource">>},
         {<<"to">>,
        <<"alicE_resume_session_state_send_message_12.909009@localhost">>},
         {<<"xml:lang">>,<<"en">>},
         {<<"type">>,<<"chat">>}],
        [{xmlel,<<"body">>,[],[{xmlcdata,<<"msg-2">>}]},
         {xmlel,<<"delay">>,
           [{<<"xmlns">>,<<"urn:xmpp:...

Report log


pgsql_mnesia_24 / pgsql_mnesia / 2c616cd
Reports root/ big
OK: 3090 / Failed: 0 / User-skipped: 213 / Auto-skipped: 0


mysql_redis_24 / mysql_redis / 2c616cd
Reports root/ big
OK: 3073 / Failed: 0 / User-skipped: 230 / Auto-skipped: 0


elasticsearch_and_cassandra_24 / elasticsearch_and_cassandra_mnesia / 2c616cd
Reports root/ big
OK: 1861 / Failed: 0 / User-skipped: 327 / Auto-skipped: 0


mssql_mnesia_24 / odbc_mssql_mnesia / 2c616cd
Reports root/ big
OK: 3101 / Failed: 1 / User-skipped: 213 / Auto-skipped: 0

pep_SUITE:pep_tests:unsubscribe_after_presence_unsubscription
{error,
  {{badmatch,
     [{xmlel,<<"message">>,
        [{<<"from">>,
        <<"alice_unsubscribe_after_presence_unsubscription_14.633715@localhost">>},
         {<<"to">>,
        <<"bob_unsubscribe_after_presence_unsubscription_14.633715@localhost/res1">>},
         {<<"type">>,<<"headline">>}],
        [{xmlel,<<"event">>,
           [{<<"xmlns">>,
           <<"http://jabber.org/protocol/pubsub#event">>}],
           [{xmlel,<<"items">>,
            [{<<"node">>,<<"kMwiS0hxheGtjGyrjxXSmQ==">>}],
            [{xmlel,<<"item">>,
               [{<<"id">>,<<"salmon">>}],
               [{xmlel,<<"entry">>,
                  [{<<"xmlns">>,
                  <<"http://www.w3.org/2005/Atom">>}],
                  []}]}]}]},
         {xmlel,<<"headers">>,
           [{<<"xmlns">>,<<"http://jabber.org/protocol/shim">>}],
           []}]}]},
   [{pep_SUITE,'-unsubscribe_after_presence_unsubscription/1-fun-0-',2,
      [{file,"/home/circleci/app/big_tests/tests/pep_SUITE.erl"},
       {line,384}]},
    {escalus_story,story,4,
      [{file,
         "/home/circleci/app/big_tests/_build/default/lib/escalus/src/escalus_story.erl"},
       {line,72}]},
    {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1783}]},
    {test_server,run_test_case_eval1,6,
      [{file,"test_server.erl"},{line,1292}]},
    {test_server,run_test_case_eval,9,
      [{file,"test_server.erl"},{line,1224}]}]}}

Report log


riak_mnesia_24 / riak_mnesia / 2c616cd
Reports root/ big
OK: 1713 / Failed: 1 / User-skipped: 328 / Auto-skipped: 0

mod_ping_SUITE:server_ping_kill:server_ping_pong
{error,{{badmatch,[{[<<"localhost">>,mod_ping,ping_response],
          {expected_diff,5},
          {before_story,5},
          {after_story,9}}]},
    [{escalus_mongooseim,post_story_check_metrics,1,
               [{file,"/home/circleci/app/big_tests/_build/default/lib/escalus/src/escalus_mongooseim.erl"},
                {line,74}]},
     {escalus_mongooseim,maybe_check_metrics_post_story,1,
               [{file,"/home/circleci/app/big_tests/_build/default/lib/escalus/src/escalus_mongooseim.erl"},
                {line,51}]},
     {escalus_story,story,4,
            [{file,"/home/circleci/app/big_tests/_build/default/lib/escalus/src/escalus_story.erl"},
             {line,75}]},
     {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1783}]},
     {test_server,run_test_case_eval1,6,
            [{file,"test_server.erl"},{line,1292}]},
     {test_server,run_test_case_eval,9,
            [{file,"test_server.erl"},{line,1224}]}]}}

Report log

@@ -277,7 +277,7 @@ jobs:
- run:
name: Run Small Tests
command: |
SKIP_AUTO_COMPILE=true KEEP_COVER_RUNNING=1 ./tools/test.sh -p small_tests -s true -e true
SKIP_AUTO_COMPILE=true KEEP_COVER_RUNNING=1 REBAR_CT_EXTRA_ARGS=" -c " ./tools/test.sh -p small_tests -s true -e true
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmmm... what about github actions?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we gather cover for github actions?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Huh, good question, no idea 🤔

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

probably no, or at least not codecov service (so it does not cause interference).

I could add it though:

  • it would make scripts more similar in feature set
  • it would make execution a bit slower :)

I could also add an env-var COVER_ENABLED instead of that REBAR_CT_EXTRA_ARGS and maybe set it explicitly to 0 on github actions.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done ;)
Now we just use COVER_ENABLED=false to be passed into tools/test.sh.
I also cleaned other env vars in circleci config.

@mongoose-im
Copy link
Collaborator

mongoose-im commented Nov 3, 2021

small_tests_24 / small_tests / 09cd99d
Reports root / small


internal_mnesia_24 / internal_mnesia / 09cd99d
Reports root/ big
OK: 1586 / Failed: 0 / User-skipped: 297 / Auto-skipped: 0


small_tests_23 / small_tests / 09cd99d
Reports root / small


dynamic_domains_pgsql_mnesia_24 / pgsql_mnesia / 09cd99d
Reports root/ big
OK: 2721 / Failed: 0 / User-skipped: 186 / Auto-skipped: 0


dynamic_domains_mysql_redis_24 / mysql_redis / 09cd99d
Reports root/ big
OK: 2704 / Failed: 0 / User-skipped: 203 / Auto-skipped: 0


dynamic_domains_pgsql_mnesia_23 / pgsql_mnesia / 09cd99d
Reports root/ big
OK: 2721 / Failed: 0 / User-skipped: 186 / Auto-skipped: 0


ldap_mnesia_24 / ldap_mnesia / 09cd99d
Reports root/ big
OK: 1483 / Failed: 0 / User-skipped: 400 / Auto-skipped: 0


dynamic_domains_mssql_mnesia_24 / odbc_mssql_mnesia / 09cd99d
Reports root/ big
OK: 2721 / Failed: 0 / User-skipped: 186 / Auto-skipped: 0


elasticsearch_and_cassandra_24 / elasticsearch_and_cassandra_mnesia / 09cd99d
Reports root/ big
OK: 1861 / Failed: 0 / User-skipped: 327 / Auto-skipped: 0


mysql_redis_24 / mysql_redis / 09cd99d
Reports root/ big
OK: 3073 / Failed: 0 / User-skipped: 230 / Auto-skipped: 0


pgsql_mnesia_24 / pgsql_mnesia / 09cd99d
Reports root/ big
OK: 3090 / Failed: 0 / User-skipped: 213 / Auto-skipped: 0


mssql_mnesia_24 / odbc_mssql_mnesia / 09cd99d
Reports root/ big
OK: 3090 / Failed: 0 / User-skipped: 213 / Auto-skipped: 0


pgsql_mnesia_23 / pgsql_mnesia / 09cd99d
Reports root/ big
OK: 3090 / Failed: 0 / User-skipped: 213 / Auto-skipped: 0


riak_mnesia_24 / riak_mnesia / 09cd99d
Reports root/ big
OK: 1735 / Failed: 1 / User-skipped: 328 / Auto-skipped: 0

sm_SUITE:parallel:messages_are_properly_flushed_during_resumption
{error,
  {{badmatch,
     {error,
       {connection_step_failed,
         {#Fun<sm_SUITE.11.118723036>,
          {client,
            <<"alicE_messages_are_properly_flushed_during_resumption_0.38394@localhost">>,
            escalus_tcp,<0.17304.1>,undefined,
            [{username,
               <<"alicE_messages_are_properly_flushed_during_resumption_0.38394">>},
             {server,<<"localhost">>},
             {password,<<"matygrysa">>},
             {stream_management,true},
             {host,<<"localhost">>},
             {stream_id,<<"210cb55f895fead0">>}]},
          [{compression,[<<"zlib">>]},
           {starttls,true},
           {stream_management,true},
           {advanced_message_processing,true},
           {client_state_indication,false},
           {sasl_mechanisms,[<<"SCRAM-SHA-256">>,<<"PLAIN">>]},
           {caps,undefined}]},
         {timeout,get_resumed}}}},
   [{sm_SUITE,'-messages_are_properly_flushed_during_resumption/1-fun-1-',
      3,
      [{file,"/home/circleci/app/big_tests/tests/sm_SUITE.erl"},
       {line,1230}]},
    {escalus_story,story,4,
      [{file,
         "/home/circleci/app/big_tests/_build/default/lib/escalus/src/escalus_story.erl"},
       {line,72}]},
    {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1783}]},
    {test_server,run_test_case_eval1,6,
      [{file,"test_server.erl"},{line,1292}]},
    {test_server,run_test_case_eval,9,
      [{file,"test_server.erl"},{line,1224}]}]}}

Report log


ldap_mnesia_23 / ldap_mnesia / 09cd99d
Reports root/ big
OK: 1483 / Failed: 0 / User-skipped: 400 / Auto-skipped: 0

@@ -612,13 +619,7 @@ export TEST_HOSTS="${TEST_HOSTS-$TEST_HOSTS_DEFAULT}"
export BUILD_TESTS="$BUILD_TESTS"
export BUILD_MIM="$BUILD_MIM"
export TLS_DIST="$TLS_DIST"
# Pass extra arguments from tools/test_runner/selected-tests-to-test-spec.sh
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

moved it above, so we have all exports in one place

@mongoose-im
Copy link
Collaborator

mongoose-im commented Nov 4, 2021

small_tests_24 / small_tests / ed9a470
Reports root / small


internal_mnesia_24 / internal_mnesia / ed9a470
Reports root/ big
OK: 1586 / Failed: 0 / User-skipped: 297 / Auto-skipped: 0


small_tests_23 / small_tests / ed9a470
Reports root / small


ldap_mnesia_24 / ldap_mnesia / ed9a470
Reports root/ big
OK: 1483 / Failed: 0 / User-skipped: 400 / Auto-skipped: 0


dynamic_domains_pgsql_mnesia_24 / pgsql_mnesia / ed9a470
Reports root/ big
OK: 2733 / Failed: 4 / User-skipped: 186 / Auto-skipped: 0

sm_SUITE:parallel_manual_ack_freq_1:resume_session_state_send_message
{error,
  {{assertion_failed,assert_many,true,
     [#Fun<sm_SUITE.26.118723036>,#Fun<sm_SUITE.26.118723036>,
      #Fun<sm_SUITE.26.118723036>],
     [{xmlel,<<"presence">>,
        [{<<"from">>,
        <<"alicE_resume_session_state_send_message_85.803797@domain.example.com/escalus-default-resource">>},
         {<<"to">>,
        <<"alice_resume_session_state_send_message_85.803797@domain.example.com/escalus-default-resource">>},
         {<<"xml:lang">>,<<"en">>}],
        []},
      {xmlel,<<"message">>,
        [{<<"from">>,
        <<"bOb_resume_session_state_send_message_85.755614@domain.example.com/escalus-default-resource">>},
         {<<"to">>,
        <<"alicE_resume_session_state_send_message_85.803797@domain.example.com">>},
         {<<"xml:lang">>,<<"en">>},
         {<<"type">>,<<"chat">>}],
        [{xmlel,<<"body">>,[],[{xmlcdata,<<"msg-1">>}]},
         {xmlel,<<"delay">>,
           [{<<"xmlns">>,<<"urn:xmpp:delay">>},
          {<<"stamp">>,<<"2021-11-04T10:16:25.880630Z">>},
          {<<"from">>,<<"domain.example.com">>}],
           [{xmlcdata,<<"SM Storage">>}]}]},
      {xmlel,<<"message">>,
        [{<<"from">>,
        <<"bOb_resume_session_state_send_message_85.755614@domain.example.com/escalus-default-resource">>},
         {<<"to">>,
        <<"alicE_resume_session_state_send_message_85.803797@domain.example.com">>},
         {<<"xml:lang">>,<<"en">>},
         {<<"type">>,<<"chat">>}],
        [{xmlel,<<"body">>,[],[{xmlcdata,<<"msg-2">>}]},
...

Report log

sm_SUITE:parallel_manual_ack_freq_1:resume_session_state_stop_c2s
{error,{thrown,{timeout,msg}}}

Report log

sm_SUITE:parallel_manual_ack_freq_1:resume_session_state_send_message
{error,
  {{assertion_failed,assert_many,true,
     [#Fun<sm_SUITE.26.118723036>,#Fun<sm_SUITE.26.118723036>,
      #Fun<sm_SUITE.26.118723036>],
     [{xmlel,<<"presence">>,
        [{<<"from">>,
        <<"alicE_resume_session_state_send_message_98.541232@domain.example.com/escalus-default-resource">>},
         {<<"to">>,
        <<"alice_resume_session_state_send_message_98.541232@domain.example.com/escalus-default-resource">>},
         {<<"xml:lang">>,<<"en">>}],
        []},
      {xmlel,<<"message">>,
        [{<<"from">>,
        <<"bOb_resume_session_state_send_message_98.486467@domain.example.com/escalus-default-resource">>},
         {<<"to">>,
        <<"alicE_resume_session_state_send_message_98.541232@domain.example.com">>},
         {<<"xml:lang">>,<<"en">>},
         {<<"type">>,<<"chat">>}],
        [{xmlel,<<"body">>,[],[{xmlcdata,<<"msg-1">>}]},
         {xmlel,<<"delay">>,
           [{<<"xmlns">>,<<"urn:xmpp:delay">>},
          {<<"stamp">>,<<"2021-11-04T10:16:38.631664Z">>},
          {<<"from">>,<<"domain.example.com">>}],
           [{xmlcdata,<<"SM Storage">>}]}]},
      {xmlel,<<"message">>,
        [{<<"from">>,
        <<"bOb_resume_session_state_send_message_98.486467@domain.example.com/escalus-default-resource">>},
         {<<"to">>,
        <<"alicE_resume_session_state_send_message_98.541232@domain.example.com">>},
         {<<"xml:lang">>,<<"en">>},
         {<<"type">>,<<"chat">>}],
        [{xmlel,<<"body">>,[],[{xmlcdata,<<"msg-2">>}]},
...

Report log

sm_SUITE:parallel_manual_ack_freq_1:resume_session_state_stop_c2s
{error,{thrown,{timeout,msg}}}

Report log


dynamic_domains_pgsql_mnesia_23 / pgsql_mnesia / ed9a470
Reports root/ big
OK: 2721 / Failed: 0 / User-skipped: 186 / Auto-skipped: 0


dynamic_domains_mysql_redis_24 / mysql_redis / ed9a470
Reports root/ big
OK: 2704 / Failed: 0 / User-skipped: 203 / Auto-skipped: 0


pgsql_mnesia_24 / pgsql_mnesia / ed9a470
Reports root/ big
OK: 3090 / Failed: 0 / User-skipped: 213 / Auto-skipped: 0


ldap_mnesia_23 / ldap_mnesia / ed9a470
Reports root/ big
OK: 1483 / Failed: 0 / User-skipped: 400 / Auto-skipped: 0


dynamic_domains_mssql_mnesia_24 / odbc_mssql_mnesia / ed9a470
Reports root/ big
OK: 2721 / Failed: 0 / User-skipped: 186 / Auto-skipped: 0


elasticsearch_and_cassandra_24 / elasticsearch_and_cassandra_mnesia / ed9a470
Reports root/ big
OK: 1861 / Failed: 0 / User-skipped: 327 / Auto-skipped: 0


pgsql_mnesia_23 / pgsql_mnesia / ed9a470
Reports root/ big
OK: 3090 / Failed: 0 / User-skipped: 213 / Auto-skipped: 0


mysql_redis_24 / mysql_redis / ed9a470
Reports root/ big
OK: 3073 / Failed: 0 / User-skipped: 230 / Auto-skipped: 0


mssql_mnesia_24 / odbc_mssql_mnesia / ed9a470
Reports root/ big
OK: 3090 / Failed: 0 / User-skipped: 213 / Auto-skipped: 0


riak_mnesia_24 / riak_mnesia / ed9a470
Reports root/ big
OK: 1707 / Failed: 0 / User-skipped: 328 / Auto-skipped: 0

.circleci/config.yml Outdated Show resolved Hide resolved
.circleci/config.yml Outdated Show resolved Hide resolved
@mongoose-im
Copy link
Collaborator

mongoose-im commented Nov 4, 2021

small_tests_24 / small_tests / 42c06f9
Reports root / small


internal_mnesia_24 / internal_mnesia / 42c06f9
Reports root/ big
OK: 1586 / Failed: 0 / User-skipped: 297 / Auto-skipped: 0


small_tests_23 / small_tests / 42c06f9
Reports root / small


ldap_mnesia_23 / ldap_mnesia / 42c06f9
Reports root/ big
OK: 1483 / Failed: 0 / User-skipped: 400 / Auto-skipped: 0


dynamic_domains_pgsql_mnesia_23 / pgsql_mnesia / 42c06f9
Reports root/ big
OK: 2728 / Failed: 1 / User-skipped: 186 / Auto-skipped: 0

sm_SUITE:parallel_manual_ack_freq_1:resume_session_state_stop_c2s
{error,{thrown,{timeout,msg}}}

Report log


dynamic_domains_mysql_redis_24 / mysql_redis / 42c06f9
Reports root/ big
OK: 2710 / Failed: 1 / User-skipped: 203 / Auto-skipped: 0

mam_SUITE:rdbms_async_cache_prefs_cases:messages_filtered_when_prefs_default_policy_is_roster
{error,{test_case_failed,"ASSERT EQUAL\n\tExpected []\n\tValue [ok,ok]\n"}}

Report log


ldap_mnesia_24 / ldap_mnesia / 42c06f9
Reports root/ big
OK: 1483 / Failed: 0 / User-skipped: 400 / Auto-skipped: 0


dynamic_domains_mssql_mnesia_24 / odbc_mssql_mnesia / 42c06f9
Reports root/ big
OK: 2721 / Failed: 0 / User-skipped: 186 / Auto-skipped: 0


dynamic_domains_pgsql_mnesia_24 / pgsql_mnesia / 42c06f9
Reports root/ big
OK: 2721 / Failed: 0 / User-skipped: 186 / Auto-skipped: 0


pgsql_mnesia_23 / pgsql_mnesia / 42c06f9
Reports root/ big
OK: 3108 / Failed: 1 / User-skipped: 213 / Auto-skipped: 0

jingle_SUITE:all:resp_4xx_from_sip_proxy_results_in_session_terminate
{error,
  {{assertion_failed,assert,is_iq_result,
     {xmlel,<<"iq">>,
       [{<<"from">>,<<"error.480@localhost">>},
        {<<"to">>,
         <<"alice_resp_4xx_from_sip_proxy_results_in_session_terminate_14.189459@localhost/res1">>},
        {<<"id">>,<<"9d3bcc93-bb7d-4081-9afb-c04262f1832d">>},
        {<<"type">>,<<"set">>}],
       [{xmlel,<<"jingle">>,
          [{<<"xmlns">>,<<"urn:xmpp:jingle:1">>},
           {<<"action">>,<<"session-terminate">>},
           {<<"sid">>,<<"5c3ebbad-2e9b-49e2-9020-7d8a1203f8b1">>}],
          [{xmlel,<<"reason">>,[],
             [{xmlel,<<"general-error">>,[],[]},
            {xmlel,<<"sip-error">>,
              [{<<"code">>,<<"480">>}],
              [{xmlcdata,<<"Temporarily Unavailable">>}]}]}]}]},
     "<iq from='error.480@localhost' to='alice_resp_4xx_from_sip_proxy_results_in_session_terminate_14.189459@localhost/res1' id='9d3bcc93-bb7d-4081-9afb-c04262f1832d' type='set'><jingle xmlns='urn:xmpp:jingle:1' action='session-terminate' sid='5c3ebbad-2e9b-49e2-9020-7d8a1203f8b1'><reason><general-error/><sip-error code='480'>Temporarily Unavailable</sip-error></reason></jingle></iq>"},
   [{escalus_new_assert,assert_true,2,
      [{file,
         "/home/circleci/app/big_tests/_build/default/lib/escalus/src/escalus_new_assert.erl"},
       {line,84}]},
    {jingle_SUITE,send_initiate_and_wait_for_first_iq_set,2,
      [{file,"/home/circleci/app/big_tests/tests/jingle_SUITE.erl"},
       {line,395}]},
    {jingle_SUITE,
      '-res...

Report log


pgsql_mnesia_24 / pgsql_mnesia / 42c06f9
Reports root/ big
OK: 3092 / Failed: 1 / User-skipped: 213 / Auto-skipped: 0

sm_SUITE:unacknowledged_message_hook:unacknowledged_message_hook_offline
{error,{{badmatch,false},
    [{escalus_session,stream_resumption,2,
              [{file,"/home/circleci/app/big_tests/_build/default/lib/escalus/src/escalus_session.erl"},
               {line,259}]},
     {escalus_connection,connection_step,2,
               [{file,"/home/circleci/app/big_tests/_build/default/lib/escalus/src/escalus_connection.erl"},
                {line,160}]},
     {lists,foldl,3,[{file,"lists.erl"},{line,1267}]},
     {escalus_connection,start,2,
               [{file,"/home/circleci/app/big_tests/_build/default/lib/escalus/src/escalus_connection.erl"},
                {line,144}]},
     {sm_SUITE,unacknowledged_message_hook_offline,4,
           [{file,"/home/circleci/app/big_tests/tests/sm_SUITE.erl"},
          {line,830}]},
     {sm_SUITE,unacknowledged_message_hook_common,2,
           [{file,"/home/circleci/app/big_tests/tests/sm_SUITE.erl"},
          {line,882}]},
     {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1783}]},
     {test_server,run_test_case_eval1,6,
            [{file,"test_server.erl"},{line,1292}]}]}}

Report log


elasticsearch_and_cassandra_24 / elasticsearch_and_cassandra_mnesia / 42c06f9
Reports root/ big
OK: 1861 / Failed: 0 / User-skipped: 327 / Auto-skipped: 0


mysql_redis_24 / mysql_redis / 42c06f9
Reports root/ big
OK: 3079 / Failed: 1 / User-skipped: 230 / Auto-skipped: 0

mam_SUITE:rdbms_cache_prefs_cases:messages_filtered_when_prefs_default_policy_is_never
{error,{test_case_failed,"ASSERT EQUAL\n\tExpected []\n\tValue [ok]\n"}}

Report log


mssql_mnesia_24 / odbc_mssql_mnesia / 42c06f9
Reports root/ big
OK: 3090 / Failed: 0 / User-skipped: 213 / Auto-skipped: 0


riak_mnesia_24 / riak_mnesia / 42c06f9
Reports root/ big
OK: 1707 / Failed: 0 / User-skipped: 328 / Auto-skipped: 0

Copy link
Collaborator

@NelsonVides NelsonVides left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Excellent 👌🏽

@NelsonVides NelsonVides merged commit 1ec0530 into master Nov 4, 2021
@NelsonVides NelsonVides deleted the mu-skip-cover-for-small-tests branch November 4, 2021 16:22
@Premwoik Premwoik modified the milestones: 5.1.0, 5.0.0 May 25, 2022
@Premwoik Premwoik modified the milestones: 5.0.0, 5.1.0 May 25, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants